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Abstract. Consider the Gelfand pairs (G P ,K V ) := {M Vtq x U P ,U P ) associ- 
ated with motion groups over the fields F = R, C,H with p > q and fixed q 
as well as the inductive limit p — > oo, the Olshanski spherical pair (Goo, K^). 
We classify all Olshanski spherical functions of (Goo , i^oo ) as functions on the 
cone Uq of positive semidefinite q X q-matrices and show that they appear as 
(locally) uniform limits of spherical functions of (G P ,K P ) as p — > oo. The 
latter are given by Bessel functions on U q . Moreover, we determine all posi- 
tive definite Olshanski spherical functions and discuss related positive integral 
representations for matrix Bcsscl functions. We also extend the results to the 
pairs (Mp j( j x (U p X U q ), (Up X U q )) which are related to the Cartan motion 
groups of non-compact Grassmannians. Here Dunkl-Bessel functions of type 
B (for finite p) and of type A (for p — > oo) appear as spherical functions. 



1. Introduction 

Let M Pt q := M Pj9 (F) be the vector space of p x g-matrices over one of the fields 
F = R, C, H and U p = U p (¥) the group of unitary p x p-matrices over F. Consider 
the Gelfand pairs (G p , K p ) := (M p _ q xi U p , U p ) associated with motion groups over 
F where q is fixed and p > q is increasing. The inductive limit (Goc-^oo) := 
(lim_j. G p , lim^ K p ) — (Mrx,^ x C/k,,!/^) is an Olshanski spherical pair; see [F] , 
[Ollj , [012] for the notion. The main purpose of this paper is to classify all Olshanski 
spherical functions of this pair, i.e., the continuous iCoo-biinvariant functions 4> '■ 
Geo — > C which satisfy the product formula 



(j){g)(j){h) = lim / 4>(gkh) dk for g, h G Goo (1-1) 

p-*°° J Kp 

with respect to the normalized Haar measures dk on K p . We shall obtain a classi- 
fication in terms of certain exponential functions which relies on the fact that for 
integers p > q as well for p — oo, the double coset space G p //K p may be identi- 
fied with the cone H q of positive semidefinite q x q-matrices over F, and that for 
finite p, formula (|I.1|) can be made explicit by the results of [Rlj . It is known (see 
e.g. [H], [FKj . |Rlj and references cited there) that for finite p, Bessel functions 
J M of index /i = pd/2 on the cone II g provide spherical functions of (G p , K p ); here 
d := dirriK¥ = 1, 2, 4. Moreover, it is also known that under suitable rescaling, the 
Bessel functions J M converge (locally) uniformly to exponential functions, see |011j 
and |RV2j , and references therein. From this we obtain that all Olshanski spherical 
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functions appear as (locally) uniform limits of spherical functions on (G P ,K P ) for 
p — > oo. 

The classification of those Olshanski spherical functions which are positive def- 
inite is also easily achieved. The connection between positive definite spherical 
functions of (G p ,K p ) with different values of p leads to the existence of positive 
integral representations for the involved matrix Bessel functions J p d/2 a s already 
explained by Schoenberg [S] in a general setting. Following mainly [H] and |FK] 
we shall derive explicit formulas for these integral representations of J„ in terms of 
Jft for v > fi in two different ways. A comparison of both appoaches will imply an 
identity for projections of beta distributions on matrix cones. 

Besides the Gelfand pairs (M Ptq x U p , U p ) we also consider the Cartan motion 
pairs (M p ^ q x (U p x U q ), U p x U q ) and the associated Olshanski spherical functions as 
p — > oo. For finite p, the spherical functions are now Dunkl-Bcssel functions of type 
B which converge for p — > oo to certain Dunkl-Bessel functions of type A, and these 
in turn are just the Olshanski spherical functions of (Afoo. ? x (Uco x U q ), Uoo x U q ); 
see [BF], [Du], [O], [Rl], [RVl] for the background. 

The approach to Olshanski spherical functions taken in this paper is similar 
to the one for noncompact infinite dimensional Grassmannian manifolds in RKV 
where the finite dimensional spherical functions are Heckman-Opdam hypergeomet- 
ric functions of type B which converge to hypergeometric functions of type A. In 
this case, the classification of the Olshanksi spherical functions is based on a prod- 
uct formula in [R2j . The present paper as well as [RKV] were partly motivated by 
related results in DOW I. 

Our approach via special functions does not use explicit representation theory for 
Olshanski spherical pairs in contrast to [Oil] . [012] . [F] or further related references. 
This has the slight advantage that we need not restrict to positive definite spherical 
functions first, but can deal with general spherical functions from the beginning. 

The paper is organized as follows: In Sections 2 and 3 we collect some facts about 
matrix Bessel functions and Dunkl-Bessel functions of type B and type A. The 
Gelfand pairs (M P}Q x U p , U p ) and the corresponding Olshanski spherical functions 
will then be studied in Section 4, while Section 5 is devoted to the pairs (M Piq x 
(Up X Ug),U p X Ug). In Section 6 we return to matrix Bessel functions and study 
their positive integral representations. 

2. Bessel functions on matrix cones 

In this section we collect some facts about Bessel functions on matrix cones. The 
material is mainly taken from |FKj . [H] . and |Rlj . and is in part slightly generalized. 

Let F be one of the fields M, C or M with real dimension d = 1, 2 or 4 respectively. 
Denote the usual conjugation in F by t H> t, the real part of t £ F by Dit = ^(t + t). 

For p, q £ N we denote by M P:q := M p . ? (F) the vector space of all p x g-matrices 
over F and put M q := M q (¥) Af 9ig (F) for abbreviation. Let further 

H q = H q (W) = {x£ M q (¥) :x = x*} 

be the space of Hcrmitian q x ^-matrices over F, where x* — x l denotes the usual 
involution on M q . The space H q is a real Euclidean vector space with scalar product 
(x, y) := tr(xy) and norm ||x|| = (x, a;) 1 / 2 , where tr is the trace. Notice that tr(xy) 
is real for x,y £ H q . H q is a (Euclidean) Jordan algebra with the usual Jordan 
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product x o y — \{xy + yx). The real dimension of H q is 

n := n(q,¥) := dim R H q = q + ^q(q - 1). 

We also need the complexification H~ of H q which is a Jordan algebra over C in 
the natural way. The extension of the scalar product ( • , • ) to a hcrmitian scalar 
product on H q will again be denoted by (x, y). For F = K or C, the space H q may 
be written as 

= {a + ib: a,be H q (¥)} C M q (C). 
More precisely, for F = R we have H^{R) = {a E M q (C) : a f = a}, while for F = C, 
H%(C) = M q (C); c.f. Section VIII.5 of [EE]. The complexification of the Jordan 
algebra H q (M) can be desribed as follows, c.f. Sections V.2 and VIII.5 of FK : One 
realizes matrices z € H q (M) as complex Hermitian matrices 

x y 
-y x 



e H 2q (C) 

where x € H q (C) is Hermitian and y € Skew q (C) is skew-symmetric. Then with 



J := l j J J e M 2q , the mapping 

z^-Jz= (l ~ y X ^j e Skew 2q (C) 

defines a real Jordan algebra isomorphism from H q (M) onto the Jordan algebra 
V q '■= {u G Skew2 q (C) : u* — JuJ} with Jordan product uov := ^(uJv+vJu). The 
complexification of this Jordan algebra is just Skew2 q (C) (with the same Jordan- 
product). We thus identify H q (M) and Skew2q(C) as complex Jordan algebras. 
Let 

U q := {x 2 : x € = {x*x : x G 
denote the set of all positive semidefinitc matrices in H qi and VL q its interior con- 
sisting of all strictly positive definite matrices. il q is a symmetric cone; see [FKj 
for details. On H q we use the standard partial ordering 

x <y y - x eU q . 

To define Bessel functions, we need the spherical polynomials 



$>,(x) = / A\(uxu 1 )du, x € H q 
Ju„ 



which are indexed by partitions A = (Ai > A2 > . . . > X q ) £ Nq (for short, A > 0). 
Here du denotes the normalized Haar measure of U q = U q (¥), and A\ is the power 
function 

A A (x) := A!(x) Al - A2 A 2 (x) A2 - A3 • . . . • A 9 (x) A « (x e H q ) 

with the principal minors Aj(x) of x; see |FKj . There is a renormalization Z\ = 
c\&\ with suitable constants c\ > depending on n ? such that 

{trxf = ^ Z x {x) for k € N ; (2.1) 

|A|=fc 

see Section XI.5. of |FK] where the Z\ are called zonal polynomials. By con- 
struction, the Z\ are invariant under conjugation by U q and thus depend only on 
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the eigenvalues of their argument. More precisely, for x <G H q with eigenvalues 

£ = (d 

Z x {x)=C%{£) with a = - d (2.2) 

where the C" are the Jack polynomials of index a (c.f. |FK| . [M] . |Rlj ). These 
are homogeneous of degree |A| and symmetric in their arguments. Notice that the 
zonal polynomials Z\ naturally extend to H q . 

The Bessel functions associated with the cone il q are defined as o-Fi-hyper- 
geometric series 

where the generalized Pochhammer symbol (p)\ is given by 

Ma = (/i) 2 /" with (m)X := f[(A* - ^(j - l)) Aj (a > 0). 

3=1 a 

The index /x 6 C is required to satisfy (//)" ^ for all A > 0. In this case, J M is 
holomorphic on H q . 

If q = 1, then n g = R_|_, and the Bessel function J M is independent of d with 

z 2 

where j K (z) = oFi( K + 1; ~ z2 /4) is the modified Bessel function in one variable. 
We need the following product formula for the J M from [Rlj : 

Proposition 2.1. For fieC with 5R/i > g?(<? — 1/2) and all r, s e 

J M (r 2 ) J M (s 2 ) = — / J M (r 2 + s 2 + nus + sw*r) A(I q - w*wY~ 1 

K M JB q 

where A denotes the determinant on H q , dw the Lebesgue measure on the ball 

B q := {w E M q : w*w < I q }, 

7 := d(q - 1/2) + 1, and k m := f J Bq A(I q ~ w*wY~^ dwj . 

Proof. For r, s £ H q we refer to Eq. (3.8) of [Rlj . As both sides of the equation 
are holomorphic in r, s € H^, the result is true in general. □ 

For the limit case fi = d(q — 1/2) there exists a degenerated product formula; 
see Section 3.5 of |Rlj . 

For real n > d{q — 1/2), product formula 12.11 leads to a positive (hypergroup) 
convolution structure on the Banach space Mf,(IIg) of all bounded signed Borel 
measures on II 9 , as follows: define first the convolution of point measures 

(S r * M 8s)(f) ■= — [ f(Vr 2 + s 2 + swr + rw*s) A(I - ww*)^ p dw (r.sell,) 

,(2-4) 

for / G Cb(Jlq), and then extend this convolution in a weakly continuous, bilinear 
way to M(,(II ? ). This generates a probability preserving commutative Banach al- 
gebra (Mb(ILq), *^); see |R1] for details. For /x = pd/2 with integers p > 2q, this 
convolution is just the double coset convolution associated with the Gelfand pair 
(M Pi9 x Up, Up) (see [Rlj ) which we shall consider in more detail in Section 4. 
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For s € Hq we define the continuous function 



/."(r) := U~ A rsr) (r e H%). (2.5) 



as well as the function 



€(r):=f^r)=J fl (-rs 2 r), (2.6) 

where the latter definition is in correspondence with the notion in [R1J . The fol- 
lowing facts slightly generalize results from |Rlj . 

Lemma 2.2. (1) For all r,s<E Hf, <ft(r) = #f(s). 
(2) For all r,teU q and s G H^, 

mr)mt)= I f»(z)d(6 r *,S t )(z), (2.7) 

V 



i.e., the functions fjf with s G Hr are multiplicative with respect to 



Proof. Both statements are known for r, s, t £ Tl q ; see the results for 4> s in Section 
3 of |R1) and notice for the second statement that each s £ H q may be written 
as s — s 2 with some unique s E H q . The general statements now follow easily by 
analytic continuation from H q to For this, notice that \ H q is connected, 
and that the identity theorem can be applied because Tl q has non-empty interior in 

H q . ' ' a 

3. DUNKL-BESSEL FUNCTIONS ON WEYL CHAMBERS OF TYPE B AND TYPE A 

There is a close connection between the Bessel functions on the cone U q and 
the theory of Dunkl operators associated with the root system B q , see |R1) . We 
briefly review this connection. We do not go into details of Dunkl theory, but refer 
to |BF] . |Du] , [O] and |Rlj . For a reduced root system R C IR 9 and a multiplicity 
function k : R — > [0, oo) (i.e. k is invariant under the action of the corresponding 
reflection group), we denote by Jj? the Dunkl-Bcssel function associated with R 
and k. It is obtained from the Dunkl kernel by symmetrization with respect to 
the underlying reflection group. Dunkl-Besscl functions generalize the spherical 
functions of Euclidean type symmetric spaces, which occur for crystallographic 
root systems and specific discrete values of k, see [D]. For the root system A g _i = 
{±(e, — gj) : i < j} C M 9 , the multiplicity k is a single real parameter. If k > 0, 
then according to formulas (3.22) and (3.37) of |BF] the associated Dunkl-Bessel 
function is the generalized o^b-hypergeometric function 

A>0 ' '' A ^ ' 

with 1 = (1, . . . , 1), a = 1/k. For k = £ with d = dim^F, the Dunkl-Bessel func- 
tions £ M> Jd/2(£> r l) are known to be the spherical functions of the flat symmetric 
space H q (¥) x U q (¥)/U q (¥) and therefore have the Harish-Chandra type integral 
representation 

Jf /2 (Z,T,)= f e^<^du. (3.1) 

Ju q (¥) 

See also formula (7) of [RVl], where an alterantive proof of (|3.1j) is given. 
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For the root system B q = {±ej, ±e; ± ej : i < j}, we have k = (fci,fc2) where 
ki and &2 correspond to the roots ±e^ and ±a ± respectively. The associated 
Dunkl-Bessel function is 

Jkfov) = oF?(fi; |-) with a = fx = h + (q - l)k 2 + 1/2 

where £ 2 = • ■ • , £ 2 ) and 



^ n (Ai)?|A|! C?(l) ' 



The Dunkl-Bessel function jj^ is invariant in both arguments under the action of 
the hyperoctahedral group which is generated by sign changes and permutations of 
the coordinates. For suitable multiplicity k, it is related the matrix Bessel function 
and to the functions 0£(r) = Jf_,(jsr 2 s) of Section 2, as follows: Consider the 
-B 9 -Weyl chamber 

Cf := U = ••• ,£,) G : ti > ■■■ > C, > 0} (3.2) 
For £ G G^ we denote by £ g II 9 the diagonal matrix with entries £i , . . . , £ g . Let 
fc(/M) := (/x-(%-l) + l)/2, d/2). 



Then for £, ?? € , 



J j?0M) (£>»?)= / ^(jRufu l rj)du; (3.3) 

see Section 4 of |R1) . Dunkl-Bessel functions of type B will play an important role 
in Section 5 of this paper, in connection with the study of the Olshanski spherical 
pairs (Moo,, x (Uoo x Uq), {Too x U q ). 

4. Olshanski spherical functions related to Mx>,g x C^oo 

In this section, we consider Olshanski spherical pairs associated with matrix 
cones. For a general background on Olshanski spherical pairs and their spherical 
functions we refer to Faraut [F] and Olshanski [Oil] , [012 . 

We fix the field F € {M,C, H} and the rank q G N as in Section 2. Consider 
the Gelfand pairs (G p ,K p ) with G p — M p>q X U p and K p := C/ p where Z7 P acts on 
Mp )9 by left multiplication. In all cases, G p will be regarded as a closed subgroup 
of Gp+i with K p = G p n K p+ i. Consider the inductive limits Goo := lim_> G p and 
Krx, := lim_> Xp. Then (Goo, -Koo) is an Olshanski spherical pair. A continuous 
function <j) : Goo — > C is called an Olshanski spherical function of (Goo,^oo) if <fi is 
i^oo-biinvariant and satisfies the product formula 



4>(g)(/)(h) = lim / 4>{gkh) dk for all g, h G Goo (4.1) 
Jk p 

with respect to the normalized Haar measure dk. Further, an Olshanski spherical 
function <p of (Goo, ^oo) is called positive definite if <f> is positive definite on Goo m 
the usual sense, i.e. if for all n G N, gi g n G G^ and c\ c„ G C. 



^2 c kd <l>(9kg l *) > 0. 



k.l=l 
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Positive definite continuous functions on Goo are always bounded. Furthermore, in 
our situation 

K 00 g^ 1 K 00 = KoogKao for g e Goo- 
This implies that positive definite spherical functions on are automatically 
K- valued. 

We shall now classify the Olshanski spherical functions of (GocKoo) as well as 
those which are positive definite in addition. 

For this we recapitulate that for each p, the space of double cosets G p //K p can 
be topologically identified with the cone H q via the homeomorphism 

K p (x, k)K p i— !► V x*x for x € M p ^ q , k g K p . 

In other words, if for a € II g we consider the matrix (^q \ ^ Mp.q> then the inverse 
of the above homeomorphism can be written as 

F p : Ilg -> Gp//K p , r — > K p g a K p with g a := ,I p ). 

By definition of the inductive limit topology, the mapping 

F : Yl q Gool/Koo, a i — > K^gaK^ with g a := ( ^ , loo) 

also provides a homeomorphism for p = oo. We will therefore use the agreement 
that for all integers p > q as well as for p = oo, continuous and i£p-biinvariant 
functions on G p are identified with continuous functions on H q . When doing so, 
our notations immediately imply the following 

Lemma 4.1. Let p\ > P2 > q. //a continuous function on Ii q corresponds to 
a K pi -biinvariant, positive definite function on G Pl , then it also corresponds to a 
K P2 -biinvariant, positive definite function onG P2 . Moreover, a continuous function 
on Il q corresponds to a Koo-biinvariant, positive definite function on Goo if and only 
if there exists an integer po > 1 such that for all integers p > po, it corresponds to 
a K p -biinvariant, positive definite function on G p . 

Proof. The first statement follows immediately from the definition of positive def- 
inite functions and the fact that for p\ > pi > q (where possibly p\ = oo), the 
canonical projections 

P p : G p — > G p // K p = Tlq , Pp((x, k)) := y/x*x 

satisfy P Pi \g P2 = P P2 - The remaining part of the second statement follows also 
easily by these arguments. □ 

We next turn to the (not necessarily positive definite) spherical functions of 
{G p ,K p ) for finite p > q. We know from Lemma [2.21 that the functions ff d ^ 2 of 
Section 2 with s £ are spherical. On the other hand, using results of Wolf 
|Wlj in combination with an integral representation of the matrix Bessel functions 
ff /2 (see Eq. (3.4) of [Rl] and Propos. XVI.2.3 of [FK]), we obtain the following 
classification: 

Theorem 4.2. Letp > q be finite. Then {ff d ^ 2 : s E Hq} is the set of all spherical 
functions of (G p , K p ). Moreover, the set of positive definite spherical functions of 
(G p , K p ) is given by {ff d/2 : s G U q }. 
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Proof. According to }Wlj (see also |W2j ). the complete set of spherical functions 
of (G p , K p ) can be described as follows: 

Consider M M as a real vector space of dimension dpq with Euclidean scalar 
product 

(x\y) := mr(x*y) 

and extend this form in a bilinear way to the complexification M pq of M p 9 . Then 
it is easily checked that for each y £ M^ ql the function 

4>y{x,v) := [ e^ ux ^du {xeM p , q ,veU p ) (4.2) 

J Up 

defines a spherical function of (G p ,K p ). Moreover, by Theorem 4.4 of |Wlj . all 
spherical functions are given in this way, and by Theorem 5.4 of [Wlj . the set of 
positive-definite spherical functions is made up by those <f> y with y £ M Pjq . 

We next show that all functions of the form (|4.2[) are in fact Bessel functions 
fP d / 2 for suitable s £ H q . For this we again regard M Pt g and H q as vector spaces 
over R, and denote the complex-bilinear extension of the R-bilinear mapping 

M Pi q x M M -> M q , (x, y) i — ► x*y 

(to a mapping M~ q x M~ q —> M q ) again by x*y. For y g M^ q we then obtain 
easily that y*y £ H q . 

Now fix a matrix y £ M^ q . Then y*y £ H q , and for all r £ U q , 

fy*l 2 ( r ) = J P d/2(^ry*yr) = J pd /2(^(yr)*yr). 

We conclude from Eqs. (3.3) and (3.4) of [EE] (see also Propos. XVI.2.3 of |FK] ) 
that for all x £ M p>q , 

J pd/2 {\x*x) = jf e-^^l^du with a = € M p ,,. (4.3) 

Using the complex-bilinear and thus analytic extension above, we conclude that 
(|4.3j) remains correct for all x £ M pq , Now let r £ U q , y £ M^ q . Then 

JpdM-iyrTyr) = [ e-^^du. 
4 Ju p 

As 

(ua \yr) = ^ttr((uao)*yr) = Utr((ucror)* y) = (ua r\y), 

we obtain 

fv¥(r)= [ e- i ^° r Mdu = 4> y (a r,v) (4.4) 

J Up 

with arbitrary v £ U p . As the functions on both sides are biinvariant, and the 
r £ Uq form a set of representatives of all double cosets as descibed in the begin- 
ning of this section, the proof of the first statement of the theorem is complete. 
Moreover, equation (|4.4I) in combination with Theorem 5.4 of |W1) leads to the 
stated classification of the positive definite spherical functions. □ 

We mention that the statement about the positive definite spherical functions 
above can be also obtained by hypergroup methods from Theorem 3.12 of |Rlj in 
combination with results of [J] . 
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We now turn to the case p = oo. The Olshanski spherical functions of (Goo , Koa) 
can be characterized as follows: 

Lemma 4.3. A continuous Koo-biinvariant function (p ■ Goo — > C is Olshanski 
spherical if and only if the continuous function </>(&) := 4>{gb) on IL q satisfies the 
product formula 

4>{a) ■ <t>(b) = 4>(y / a 2 + b 2 ), a,b£U q . (4.5) 

Proof. Let ^ be a continuous i-Too-biinvariant function on Goo and <p € C(H q ) 
as described in the lemma. Then, by (|4.1I) and the product formula (|2.4[) . tj> is 
Olshanski spherical iff <j> satisfies 

$(a) ■ 4>(b) = lim — !— [ 4>(Va 2 + b 2 + awb + bw*a) ■ A(I - w*w) pd/2 ^ dw 

P^oo K pd/2 J Bq 

(4.6) 

for a, b £ H q with 7 = d(q — 1/2) + 1. The probability measures 

K^ /2 -A(i- w * w y d ^dw 

are compactly supported in B q and tend weakly to the point measure 5q for p — > 00. 
Therefore (|4.6j) is equivalent to 

4>(a) ■ 4>{b) = 4>(y/a 2 + b 2 ) 
as claimed. □ 

We remark that precise estimates for the order of convergence of the probability 
measures «~^/ 2 ■ A(7 — w*w) pd / 2 ~ 1 dw are given in |V2] . 
We now solve the functional equation (14.51) : 



Lemma 4.4. A continuous function <j) £ C(JA q ) with 0(0) = 1 satisfies l{4-5\ ) if and 



only if there exists some b £ such that 



4>{a) = exp(-(a 2 , b)) =: ipb(a), «ell,. 

Proof. Clearly, all ipb satisfy (|4.5p . Conversely, if a function <j> £ C(H q ) satisfies 
(|4.5|) . then ip(a) := <j)(y/a) satisfies tp(a)ip(b) — ip(a + b) for a, b £ U q and thus, 



because of ^(0) = 1 and continuity, ip(a) ^ for all a £ IL q . Each a £ H q can 
be written as a = d — cl q for some d £ Tl q and some c £ [0, 00 [, and it is easily 
checked that ip(a) := ip(d)/ip(cl q ) defines a well-defined function ijj £ C(H q ) with 
^{a)ip(b) = i/j(a + b) for a,b £ H q . The assertion now follows by the well-known 
characterization of the exponential function on a Euclidean space by its functional 
equation. □ 

The preceding lemmata immediately lead to the following characterization of the 
Olshanski spherical functions of (Goo, Koa). 

Theorem 4.5. A continuous Koo-biinvariant function <fi : Goo — > C is Olshanski 
spherical if and only if 

4>{9a) = ifa(a) = exp(-(a 2 ,6)) (a £ U q ) 

for some b £ H q . 

We next investigate which Olshanski spherical functions of (Goo, Koo) appear as 
limits of spherical functions on (G p , K p ) for p — » 00. For this we employ the known 
convergence of J^fiy) to e _tr ^ for fx — > 00: 
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Lemma 4.6. For y £ H q . 

lim Ju(/iy) = e~ tr ^ locally uniformly. 

On the cone Tl q , this convergence is even uniform. More precisely, there exists a 
constant C = C'(q,¥) such that 

\JM - e~ tr ^\ < C/n for all y£U q ,fi> 2q. 

Proof. For the second statement we refer to Proposition 3.2 of [RV2j; see also 
Lemma 4.8 of [Ollj or references cited there for the uniform convergence. 

The first statement can be obtained either from the integral representation (3.12) 
of |R1] for Jfj,, or by power series expansion as in the proof of Proposition 3.5 in 
|RV2j . We outline the second approach: Using the expansions (|2.3p and (|2.1[) , we 
have 



where by Lemma 3.4 of |RV2j 



1 ' 



(m)a 

It is easily checked that the series 



X:0 |A|!V( M)A 



A>0 1 1 

converges absolutely and locally uniformly for y £ H q (c.f. Section XV. 1 of [FK] 
and our normalization of the Z\ instead of the <£>a there). This immediately leads 
to the locally uniform convergence of order 1/fM. □ 

We conclude from Lemma[05]that for all y £ H q , b £ H q and for fi = pd/2 — > oo, 
the functions fjf of Section 2 satisfy 

f^biv) = J^yby) exp(-tr(yby) / '4) = ipb/i(v) 

uniformly or locally uniformly depending on b. According to Theorem 14.21 the 
functions f£ b with b £ form the spherical functions of (G P ,K P ). Considering 
biinvariant functions on G p and Goo as functions on H q as above, we obtain 

Corollary 4.7. All Olshanski spherical functions ipb °f (Goo: -Koo) with b £ H 1 ^ 
appear as locally uniform limits of spherical functions of (G P ,K P ). 

Moreover, those Olshanski spherical functions ipb with b £ H q even appear as 
uniform limits of positive definite spherical functions of (G p ,K p ) as p — > oo. 

We finally determine the Olshanski spherical functions ipb which are positive 
definite. 

Theorem 4.8. The positive definite Olshanski spherical functions of (Goo, K^) 
are precisely given by the functions ipb with b £ Ii q . 

Proof. Assume that ipb is a positive definite Olshanski spherical function. Then ipb 
must be R- valued. From ipb (a) = exp (—(a 2 ,6)) for a € II, we infer that b £ H q . 
Moreover, as ipb is in addition bounded on LI g , it follows easily that b £ U q . In fact, 
if b would have a negative eigenvalue with eigenvector u, we could choose a matrix 
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a £ ILq with the same eigenvector u associated to the eigenvalue 1, and with all 
other eigenvalues equal to 0. It is then clear that V>h(ca) = exp (-c 2 (a 2 ,b)) tends 
to oo for c — ^ co. 

For the converse statement consider 6 G LI^. Let p > q be a fixed integer. Then 

for each integer p > p, the spherical function f^ d ^ 2 £ G(Tl q ) corresponds to a 
Kp -biinvariant positive definite function on Gp, and thus by Lemma l4.1[ to a K p - 
biinvariant positive definite function on G p . As positive dcfmitcncss is preserved 
under limits, it follows from Corollary 14.71 that ipb £ C(H q ) is a itp-biinvariant 
positive definite function on G p . This holds for all p, and therefore Lemma 14.11 
ensures that tpb is a positive definite function on Goo as claimed. □ 

In summary, the Olshanski spherical functions of (Goo, -Koo) admit a classification 
which is in complete accordance with that for finite p in Theorem 14.21 



5. Olshanski spherical functions related to Moo, g x (Uoo x U q ) 

In this section we consider the Gelfand pairs (G p , K p ) with G p = M p<q x (U p x U q ) 
and Kp := U p x U q for fixed q > 1, where the groups U p and U q act on M p>q by 
multiplication from the left and right, respectively. Consider the Olshanski spherical 
pairs (Goo, -ft'oo) with Goo := lim_> G p and := lim_^ K p . 

Again, we investigate the Olshanski spherical functions of (Goo,-f^oo)- We reca- 
pitulate first that for each p, the double coset space G p //K p can be topologically 
identified with the Weyl chamber 

cf ■= u = (a, • • • ,£,) G K 9 : ii > • • • > U > 0} 

of type B via 

K p (x, k)K p i-> a(Vx*x ) for a; £ M p>q , k £ K p 

independently of p where a stands for the ordered spectrum of a positive semi- 
definite matrix. In other words, if for £ £ C q we consider the diagonal matrix 

£ := diag(£i, . . . , £ g ) G M 9 as well as € M p . 9 , then the inverse of this homeo- 
morphism can be written as 

F p :Cf ->G P //K V , b^K p9i Kp with 9i := ,J p x I q ). 

By definition of the inductive limit topology, the F p induce a homeomorphism 
F : C q — > Gooj/Koo. Again we use the agreement that for all integers p and for 
p = oo, i^p-biinvariant continuous functions on G p will be identified with continuous 
functions on H q . When doing so, the statement of Lemma [4.11 transfers to the 
present setting without changes. 

We now turn to the classification of spherical functions: 

Proposition 5.1. The spherical functions of (G p ,K p ), considered as functions 
on the chamber C q , are precisely given by the Dunkl-Bessel functions ^ d ^ 2 (^) '■— 
Jftfp (£) if}) with rj £ C q , where [i — pd/2. Moreover, the positive definite spherical 
functions of (G p , K p ) are given by the (j$ l d ^ 2 with rj £ W . 

Proof. The first statement is known from Dunkl theory, see (Qj. The second then 
again follows from Theorem 5.4 of |Wlj . □ 
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For the case p = oo, we start with the following 
Lemma 5.2. A continuous Kao-biinvariant function <fi : — > C is Olshanski 



spherical if and only if the continuous function </>(£) := 4>(g$) on C q satisfies 



m-kv)= ~4>{a{Je + urfu^)) du, 6*7 eCf. (5.1) 

Proof. Let be a continuous i^oo-biinvariant function on Goo and <f> defined as 
above. Then, by (I4.1[) and the product formula for the spherical functions of 
(G p , K p ) (see e.g., p. 771 of |R1) V <fi is Olshanski spherical iff <j> satishes 

4>(0 ' 'Piv) = li m / / 4 > ( <J {\/ £ 2 + urj 2 u* + ^wurju* + unu*w*^)) 

p^oo Kpd/2 J Uq J Bq ^- ~ ~ ~ 

■ A(I-w*w) pd / 2 -" / dwdu, £,»jeCf. (5.2) 

As the probability measure 

• A(I -w*w) pd/2 -~< dw 
on B q tends weakly to the point measure Sq for p — > oo, (|5.2|) is equivalent to the 



condition of the lemma. □ 

We now solve the functional equation (|5.1[) by using the Dunkl-Bessel functions 
j£ of type A on the Weyl chamber 

Cf:={teW: Zi>...>Z q }DCf. 

For this we identify the space of double cosets of the Gelfand pairs (H q x U q , U q ) 
(where U q acts on H q by conjugation) with C q and recall from Section [3] that the 
spherical functions of (H q x U q ,U q ) are given by the functions x H> </^/ 2 (£> 7 ?) w ^ n 
7? G C« by P. 

Lemma 5.3. A continuous function <j) on C q with 0(0) = 1 satisfies i5.1\) if and 
only if there exists some b £ C q such that <j>(£) = J^/^i^ 2 ^) f or a ^ £ ^ • 
Here £ 2 £ C q means the vector which is obtained from £ by taking squares in each 
component. 

Proof. Let G C{Cf) with 0(0) = 1. Then satisfies |jOJ if and only if the 
function ^i>(£) := 0(V?) on satisfies 

^(0'V>(»7) = / Mr^+w/u- 1 ))^, C/€Cf. (5.3) 

In particular, for a, 6 € [0, oof we have 

^((a, . . . , o))^((6, ••.,&)) = V((a + &,-.., a + &)). 

As "0 is continuous with ip(0) = 1, this implies that there exists some c G C such 
that 

ip((a, ■ ■ . , a)) = e ca for all a G [0, co[. 
Precisely as in the proof or Lemma 14.41 it is now seen that ip can be uniquely 
extended from C q to a continuous function on C q which satisfies (|5.3I) for all 
£, 77 G C^ 1 , namely by putting 

■ • ■ , := V-(a - • • • , - 0) • e c «* for (a, . . . , e,) € c£. 
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Thus the extension ip G G(G^) is a spherical function of the Gelfand pair (H q x 
U q ,U q ). On the other hand, we know that the spherical functions of this Gelfand 
pair are precisely the Dunkl-Bessel functions £ H> Jfi^i^ib) with b G C 9 . This 
proves the claim. □ 

The preceding lemmata yield the following characterization of the Olshanski 
spherical functions. 

Theorem 5.4. A continuous K^-biinvariant function cf> : Goo — > C is Olshanski 
spherical if and only if for some dgC, 

H9d = Jd/2(e,-b) =: MO for all £ G Cf . 

We next study which Olshanski spherical functions of (Goo, -ftToo) appear as limits 
of spherical functions on (G p ,K p ) for p — > oo. For this we employ the known 
convergence of the Dunkl-Bessel functions of type B to those of type A from |RV1| : 

Lemma 5.5. For each e C, 

Inn J? M (2tfiZ, ib) = Jf /2 {e, ~b 2 ) 

locally uniformly in £ G Cq • For (£, b) G x C^, the convergence is even uniform. 

We mention that this result also follows immediately from Lemma 14.61 by taking 
the means (|3.3p and (|3.ip for the Dunkl-Bessel functions of type B and A respec- 
tively. Lemma 15.51 in combination with Proposition 15.11 and Theorem 15.41 implies 
the following result (again, we consider biinvariant functions on G p and Goo as 
functions on C q ): 

Theorem 5.6. All Olshanski spherical functions 4>b , b G C q of (Goo, ^oo) appear 
as locally uniform limits of spherical functions of (G p , K p ) for p — » oo. 

Moreover, those Olshanski spherical functions ipt, with b G C q appear even as 
uniform limits of the spherical functions of (G p , K p ) for p — > oo. 

We finally turn to the question which of the Olshanski spherical functions tpb are 
positive definite. The same argument as in the second part of the proof of Theorem 
14.81 immediately implies: 

Proposition 5.7. The functions ipf, with b € C q are positive definite Olshanski 
spherical functions of (Goo, Kqo). 

We conjecture that the converse statement holds as well. For q = 1 we are in 
fact in the situation of Theorem 14.81 and the conjecture is clear. 

For q > 2 however, there exist spherical functions ipb with b 6 C q \ M. q which 
are R-valued and bounded, and at present, we are unable to decide whether these 
functions are positive definite on Goo- 

We give an example for q = 2 and F = K: Let b = (i, —i) and £ = (£i, £2) G G.f , 
i.e. £1 > £2 > . Then in view of integral representation (|3.1[) . 



= /( ) exp (~* ' tr ^ ' U (0 -l) U 1 )) du 
= -L [ exp(-i(£ 2 - £ 2 2 )(cos 2 t - sin 2 t))dt 

^ 7F J — 7T 



1 

2^ 



cos I 



((£1 - £)(cos 2 t - sin 2 1)) dt. 
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It is unclear to us whether this function, which is a Bessel function, is positive 
definite on Goo, i.e., whether according to the version of Lemma 14. II in the context 
of this section, this function is positive definite on G p for all p > q. 

6. Positive integral representations of matrix Bessel functions 

Consider integers p 2 > Pi > q and the associated indices jik '■= Pkd/2 (k = 1,2) 
of the matrix Bessel functions. Then the functions </>^ 2 with s € II 9 as introduced in 
(|2.6|) represent positive definite biinvariant functions on G Pl , and thus by Lemma 
14.11 positive definite biinvariant functions on G P2 . Therefore, by Bochner's theorem 
for hypergroups (see [J]), which may be applied to the associated matrix Bessel 
hypergroup on II g of index p\, the function <f>^ 2 has a representation 

<fp(x)= f ^(x)du PUP2 , s (t) (ien,) (6.1) 

with some unique probability measure Vp 1 , P2 ; S G M (Tl q ). 

In this section we shall determine these measures explicitely in two different ways. 
Comparison of these results will then lead to a projection result for multivariate 
beta distributions. Our first approach was already carried out in [H] for F = K; it 
relies on the following Laplace transform identity for the Bessel functions J M which 
holds for general F; see Proposition XV. 2.1 and Corollary VII. 1.3 of [FK : 

Proposition 6.1. For all /ieC with SR/i > d(q — l)/2 and y € f2 g , 

/ J t ,(x)e~< x ^A(x)^ n/q dx = T q n (fi) ■ A(y)~" • e~ tr ^ 
with the Gamma function 

T q n (iJ,) = f e- tr ^A(x)^- n ^dx= (27r)("- 9 )/ 2 r(/i)r( M -d/2)---r(/i-( g -l)d/2). 
Jn q 

By the transformation formula for linear maps, we have for m £ H q 

[ f{mxm) dx = A(m) 2n/q [ f{x)dx. 
Jn q Jn q 

We thus obtain for /ieC with dip > d(q — l)/2, m EH q and y € O g , that 

/ J^{xm)e-^^A{xY- n/q dx^T q n {^)-A{y)-^-e-' tr{my ~ 1) ] (6.2) 
Jn q 

c.f. equation (2.5) of [H] for F = K, where a minus sign in the exponential is 
missing. We notice that (|6.2[) may be also interpreted as a well-known formula for 
the Hankel transforms of Wishart distributions; see for instance |FK| or Section 5 
of |Vlj . Using injectivity and the convolution theorem for the Laplace transform, 
we deduce from (|6.2p the following addition theorem: 

Proposition 6.2. For all fi, v E C with 5fyt > d(q— l)/2 and > d(q— l)/2 and 
all mi, ni2 € Tl q , 

^„(a(mi + ma)) ■ A (z) = (6.3) 

Mym 1 )A(yr- n / q J u ((x - y)m 2 )A(x - y)^ 9 dy. 

y£Tl q : y<x} 



r^^)r^(^) 
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Taking mi = m, rri2 — 0, x — I q (the identity matrix), and defining 

K := {y G n, : y < I,}, 

we obtain the following Sonine-type integral representation of Jn+v in terms of J^, 
which was for F = M already proven in [H] : 

Corollary 6.3. For all /1,1/eC with 5ft/i, > d(q — l)/2 and m G H q , 
•W™) = t^TWI / Mym)A( V r-^A{I q - „)""»/« dy. 
Notice that for m = this formula implies the known beta integral 

/ A(y)*- n /«A(I, - y)"-" = r o W>) „ ^ ^ (g _ 4) 
Jn' q L n{f I + v ) 

Using these notions, we define the beta distributions d/3 q;fJ , lU G M 1 (IT^) on 11^ for 
real parameters p, v > d{q — l)/2 by 

7 5 n L ^A(y)^"/«A(/ (? - d2/ 

with n = n(g,F) as in Section 2. Using Lemma |2.2f 1). we obtain the following 
explicit form of relation ()6.1j) : 

e fP „,.„+>, S»,, S»,, \ At „ 1 \ /O n ^A c m r- TT 

9' 



Corollary 6.4. for a/Z /i,i/£C > cZ(g — l)/2 and s, x € II 



Remarks. (1) We conjecture that for all real parameters v > 0, /i > cZ(q — l)/2 
and all s£ll g there is a probability measure rn^^s € M (II g ) with compact 
support such that 

4>» +v {x) = f <%{x) dm M , v , s {y) for all a: e II, 

JXia 



(2) One can easily combine Corollary 16.41 with Lemma [4.61 for v — > oo. This 
yields an explicit integral representation of the Olshanski spherical functions 
4>b of Section 4 for 6 G H q in terms of the 4>s- This formula describes ipb 
as the Hankel transform of order p of some Wishart distribution and is 
well-known, see Proposition XV. 2.1 of |FK| and Lemma 5.4. of [VI] . which 
is adapted to our notation. 



Next, we derive Corollaries 16.31 and 16.41 in a different, more geometric way as 
follows: Consider integers 1 < q < p and p > 2p. For p,p define the associated 
parameters p := pd/2 and p := pd/2 of the Bessel functions on IT 9 . ^From the fact 
that the cp't are spherical functions we conclude that for A, x G IT 9 

<ft i {x) = J exp(i-mr(^x,0)u(^jyjdu 

with (x, 0) G M q ,p and G M p>q (see also |Rlj ). Now let w G Bp be the upper 

/ w * \ 

left p x n-block of u. i.e. u = .In the following, C\. CV C3 are constants 
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depending on q,p,p, d. We first conclude from Lemma 2.1 of }R2j that for p > 2p, 
<ft(x)=CiJ exp(i-mr((x 7 0)w Q)) ■ A{I - w*w) pd/2 - d{ P- 1/2} - 1 dw 

where now (x, 0) € M 9 ^ and G Mp,q- Further, integration with respect to 

polar coordinates on Mp according to [FT] (see also p. 759 of [Rl] ) gives 



f(w)dw=C 2 / f(uVr) ■ Airf/^dudr. 

Ms JQa JUi 



Therefore 



• A(r) rf / 2 " 1 A(/ - r ypd/2-d(P-i/2)-i dr _ 



By the definition of </>^, we obtain 



A' 

l(x) = C 3 [ ct>l (r) {x)A{r) d / 2 - l A{I - r )pd/2-d(p-i/2)-i dr 



with A(r) := y(A,0)r Q en,. 

Putting x = and using (|6.4|) for p instead of q, we finally obtain from the 
definition of n that 

' / 
n 



<^( X ) = / l <l>\(r)( X ) d Pp;pd/2,d(p-p)/2( r )- (6.5) 



We now compare equation (|6.5|) with Corollary 16.41 for p = pd/2, v = (p — p)d/2 
and arbitrary x € Tl q . By the injectivity of the Hankel transform of index p on Tl q 
and by analytic continuation with respect to p and v, we shall obtain the following 
projection result for multivariate beta distributions: 

Proposition 6.5. For integers p > q > 1, consider the projection 

T/ien /or a/Z real-valued parameters p,v > d(p — l)/2, £/ie measure j3 q -^. M is the 
push-forward of the measure j3p-^. u under Pp >q , 

Pp,q\Pp;H,u) = fiq\n,v 

Proof. Assume first that // = dp/2 and v = — p)/2 for some integer p > 2p. In 
this case, the statement follows from the preceding arguments. 
Now consider integers r > p > q and p > 2r. We obtain 

Pq;dr/2,d(p-r)/2 = Pr,q(Pr;dr/2,d(p-r)/2) = Pp,q ° Pr,p(ftr;dr /2,d(p-r) /%) 
— Pp,q{Pp;dr/2,d(p-r)/2)- 

This equality means that for the parameters p — dr/2 and v — d(j> — r)/2 with 
integers p, r with r > p and p > 2r, we have for all bounded continuous functions 
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/ E C b (n^) the identity 

/ f(P(x)) dfewix) = [ f(y)d^(y). (6.6) 

By definition of the beta distributions, both sides of this identity are, for fixed /, 
analytic in the variables fi, v g C with #tfi, > d(p — l)/2. Precisely as in |Rlj . it 
is easily checked that both sides also satisfy the exponential growth conditions of 
Carlson's Theorem (p. 186 of [T]). This yields that (|6.6[) holds for general \x and 
v, which finishes the proof. □ 

We mention that Proposition 16.51 can also be derived, at least for F = R and 
/i = p/2 and v = r/2 with suitable integers p, r, from the construction of the 
multivariate beta distributions in statistics; see Section 10.2 of [Fa] and references 
cited there. Let us sketch this approach: we consider M Pt $- and M r< p -valued 
independent random variables X and Y respectively such that all entries of X and 
Y are i.i.d. standard-normal distributed. Then S := X l X, T := Y l Y ', and S+T are 
lip -valued and Wishart-distributed. Now form the unique lower triangular matrix 
C with nonnegative entries satisfying CC* = S + T. It is well-known (Section 10.2 
of jFaj and references cited there) that L := C _1 S'(C t )~ 1 is a IP- -valued variable 
with distribution /% ; p/2,r/2- 

Now consider some integer 1 < q < p. For any matrix A £ Mp we denote by A 
its upper left q x q-block. It is now easily checked from the triagular structure of 
C and C' 1 that 

with CC* —S + T. This observation readily leads to a further proof of Proposition 
16.51 for /i = p/2 and v = r/2. Again, application of Carlson's theorem implies the 
general result. 
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